Methods of pose estimation of three-dimensional bone models in surgical planning a total ankle replacement

ABSTRACT

A computer process including receiving first patient bone data of a patient leg and foot in a first pose, the first pose comprising a position and orientation of the patient leg relative to the patient foot as defined in the first patient bone data. The computer process may further include receiving second patient bone data of the patient leg and foot in a second pose, the second pose comprising a position and orientation of the patient leg relative to the patient foot as defined in the second patient bone data. The computer process may further include generating a 3D bone model of the patient leg and foot. Finally, the computer process may include modifying the 3D bone model of the patient leg and foot such that the plurality of 3D bone models are reoriented into a third pose that matches the second pose.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119 to U.S.Provisional Patent Application No. 62/500,823, which was filed May 3,2017, entitled “METHODS OF LATERAL AND ANTEROPOSTERIOR POSE ESTIMATIONOF 3D BONE MODELS IN SURGICAL PLANNING A TOTAL ANKLE REPLACEMENT,” andis hereby incorporated by reference in its entirety into the presentapplication.

TECHNICAL FIELD

Aspects of the present disclosure involve methods of lateral andanterior-posterior pose estimation of bone models in surgical planning atotal ankle replacement, and, more particularly, involve methods ofmapping weight bearing conditions of a foot from standing X-ray imagesto bone models generated from medical images of a supine patient.

BACKGROUND

Total ankle replacement (“TAR”) procedures involve replacement of theankle joint with an artificial implant that is designed to treat aparticular condition, such as arthritis or fracture of a bone formingthe joint. A conventional TAR procedure may include scanning the damagedfoot and leg of the patient with medical imaging machine (e.g., CTmachine, MRI machine) while the patient is in a supine position. Theindividual bones in each of the scans or images of the foot and leg arethen segmented. A three-dimensional (“3D”) bone model of the bones isgenerated from the segmented images, and then the surgeon may plan thesurgical procedure using the patient specific 3D bone models. Surgicalplanning may include determining implant size and position, resectiondepths and positions relative to the bones, and surgical approaches,among other parameters. Once planning is complete, the surgery is thenperformed according to the plan.

One particular error-factor in TAR procedures is valid ankle poseestimation during the surgical planning steps of the procedure given theimage scans forming the basis of the 3D bone models are not performedunder weight bearing conditions. More particularly, the image scansperformed on a non-standing, supine patient depict the bones of the footand leg (e.g., tibia, fibula, talus, calcaneus) in an un-weighted stateor condition. That is, the weight of the patient body is not acting onthe bones of the leg and foot during the imaging scans. Thus, the 3Dmodels of the bones of the leg and foot are modeled as if the bones areun-weighted. In this way, any surgical planning that takes place basedon 3D models does not take into account a standing or weighted positionof the bones relative to each other or relative to the floor. This canresult in less than desirable surgical outcomes.

Accordingly, there is a need in the art for system and methods thataddress these shortcomings, among others.

SUMMARY

Aspects of the present disclosure are directed to improving TARprocedures and planning for the same by providing methods of mappingweight bearing conditions of a foot from standing X-ray images to bonemodels generated from medical images (e.g., computed tomography (“CT”)images, magnetic resonance images (“MRI”), among others) of a supinepatient.

In certain instances, the method may include the following steps: (1)surface projection of 3D bone model to form 2D image of bone model:project the surface of the 3D bone model (generated from non-standing CTimages) in lateral and anteroposterior views to form 2D images of bonemodel; (2) contour extraction of X-ray images: segment the object regionin the X-ray images in lateral and anteroposterior views as defined bythe bone boundary; (3) shape match (1) and (2): register or map the 2Dimages of the bone models with the segmented X-ray images in one or bothof the lateral and anteroposterior views; (4) pose update: use the pointcorrespondences from the shape matching step (3) to update pose of the2D images of the bone model; (5) iterate: repeat steps (1) to (4) untilconvergence.

Aspects of the present disclosure may involve one or more tangiblecomputer-readable storage media storing computer-executable instructionsfor performing a computer process on a computing system. In certaininstances, the computer process may include: receiving first patientbone data of a patient leg and foot in a first pose, the first patientbone data generated via a first imaging modality, the first pose mayinclude a position and orientation of the patient leg relative to thepatient foot as defined in the first patient bone data. The computerprocess may further include receiving second patient bone data of thepatient leg and foot in a second pose, the second patient bone datagenerated via a second imaging modality that may be different from thefirst imaging modality, the second pose may include a position andorientation of the patient leg relative to the patient foot as definedin the second patient bone data. The computer process may furtherinclude generating a three-dimensional (3D) bone model of the patientleg and foot from the first patient bone data, the 3D bone model mayinclude a plurality of 3D bone models arranged in the first pose. And,the computer process may further include modifying the 3D bone model ofthe patient leg and foot such that the plurality of 3D bone models arereoriented into a third pose that matches a particular arrangement ofbones in the patient leg and foot in the second pose.

In certain instances, the first imaging modality may be computedtomography.

In certain instances, the second imaging modality may be X-ray.

In certain instances, the first pose may include a non-standing positionand orientation of the patient leg relative to the patient foot.

In certain instances, the second pose may include a standing positionand orientation of the patient leg relative to the patient foot.

In certain instances, the modifying the 3D bone model may includecausing first bone contour lines of the plurality of 3D bone models toalign with second bone contour lines of the second patient bone data.

In certain instances, the one or more tangible computer-readable storagemedia may further include importing the 3D bone model and the secondpatient bone data into a common coordinate system.

In certain instances, the second patient bone data may include a lateralX-ray image of the patient leg and foot in the second pose, a medialX-ray image of the patient leg and foot in the second pose, and ananteroposterior X-ray image of the patient leg and foot in the secondpose.

In certain instances, the modifying the 3D bone model may includealigning the plurality of 3D bone models with corresponding bones of thepatient leg and foot in the second patient bone data, wherein thealigning may be done in lateral, medial, and anteroposterior views ofthe plurality of 3D bone models so as to match the orientation of thepatient leg and foot in the lateral X-ray image and the anteroposteriorX-ray image.

In certain instances, the modifying the 3D bone model of the patient legand foot may be performed manually.

In certain instances, the modifying the 3D bone model of the patient legand foot may be performed automatically.

In certain instances, the modifying the 3D bone model of the patient legand foot may be performed automatically by positionally matchinglandmarks in the plurality of 3D bone models and the second patient bonedata.

In certain instances, the first patient bone data and the second patientbone data are the results of two different imaging events.

In certain instances, the second imaging modality may be X-ray, and thesecond patient bone data may include X-ray images, the computer processfurther may include: segmenting bones of the patient leg and foot in theX-ray images; and generating bone contour lines along a perimeter of atleast some of the bones in the X-ray images.

In certain instances, the one or more tangible computer-readable storagemedia may further include: generating a plurality of poses for each ofthe plurality of 3D bone models; generating a plurality oftwo-dimensional (2D) projections from the plurality of poses for each ofthe plurality of 3D bone models; and comparing the bone contour lines tothe plurality of 2D projections, and identifying particular 2Dprojections from the plurality of 2D projections that most closely matchthe bone contour lines.

In certain instances, the one or more tangible computer-readable storagemedia may further include: arranging the plurality of 3D bone modelsaccording to particular orientations of the particular 2D projectionsassociated with each of the bones.

In certain instances, the one or more tangible computer-readable storagemedia may further include: preoperatively planning a total anklereplacement procedure using the plurality of 3D bone models beingreoriented into the third pose.

In certain instances, the one or more tangible computer-related storagemedia may further include: limiting a number of the plurality of posesthat are generated to only such poses that are permissible givenbio-kinematics of the bones making up the plurality of 3D bone models.

Aspects of the present disclosure may involve a system for processingpatient data. In certain instances, the system may include: a networkinterface configured to receive one or more sets of patient data; aprocessing device in communication with the network interface; and acomputer-readable medium in communication with the processing deviceconfigured to store information and instructions that, when executed bythe processing device, performs the operations of: receiving firstpatient data may include at least one two-dimensional (2D) image of apatient leg and foot in a weighted pose; receiving second patient datamay include computed tomography (CT) images of the patient leg and footin a non-weighted pose, the first patient data and the second patientdata being the result of separate imaging events; generating athree-dimensional (3D) bone model of the patient leg and foot from theCT images, the 3D bone model may include a plurality of 3D bone modelsrepresenting individual bones of the patient leg and foot; andrearranging the plurality of 3D bone models to mimic the weighted poseof the patient leg and foot in the at least one 2D image.

In certain instances, the system may further include: generating aplurality of 2D projections of poses of the plurality of 3D bone models;comparing the plurality of 2D projections to contour lines outliningperimeters of bones of the patient leg and foot in the at least one 2Dimage; and identifying particular 2D projections from the plurality of2D projections that best-fit a shape and size of the contour lines.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof necessary fee.

Example embodiments are illustrated in referenced figures of thedrawings. It is intended that the embodiments and figures disclosedherein are to be considered illustrative rather than limiting.

FIG. 1A through 1B are flowcharts depicting an exemplary method of poseestimation of a 3D bone model in a total ankle replacement procedure.

FIG. 2 is an overhead view of a patient laying supine on an imagingtable proximate an image scanning machine.

FIG. 3 is a front view of a patient standing and having a medial x-rayof the foot.

FIG. 4 is a side view of the patient standing and having ananteroposterior x-ray of the foot.

FIG. 5A is an example medial or lateral x-ray view of the patient's footin a weighted condition.

FIG. 5B is an example anteroposterior x-ray view of the patient's footin the weighted condition.

FIG. 6 is a sagittal image slice of a plurality of image slices taken ofthe patient's foot with the patient in the supine position (non-weightedcondition), where the bones are segmented.

FIG. 7 is a lateral view of a three-dimensional bone model constructedfrom the plurality of image slices of the patient laying supine.

FIG. 8 is a lateral view of a 3D bone model overlaid on an X-ray of apatient's foot and leg bones, where X-ray depicts a standing pose of thefoot and leg, and the 3D bone model depicts a non-standing pose of thefoot and leg.

FIG. 9 is the lateral view of the 3D bone model and X-ray of FIG. 8,except the 3D bone model of the foot has been roughly aligned with theX-ray image.

FIG. 10 is the lateral view of the 3D bone model and X-ray of FIG. 8,except the individual bones of the 3D bone model of the foot and leg(tibia, fibula) have been rotated and/or translated so as to align withthe bones of the foot in the X-ray image.

FIGS. 11A-11C are flowcharts depicting an exemplary method of poseestimation of a 3D bone model in a total ankle replacement procedure.

FIG. 12A depicts a lateral-to-medial X-ray image of a patient's foot,segmented bones from the X-ray image, and a perimeter contour linedefined from the segmented bones.

FIG. 12B depicts a medial-to-lateral X-ray image of a patient's foot,segmented bones from the X-ray image, and a perimeter contour linedefined from the segmented bones.

FIGS. 13A, 13B, and 13C depict an object, representing an individualbone of a 3D bone model of a patient's foot, in three different poses,respectively.

FIG. 14A depicts a 2D projection of the outer boundary of the object inthe pose as shown in FIG. 13A.

FIG. 14B depicts a 2D projection of the outer boundary of the object inthe pose as shown in FIG. 13B.

FIG. 14C depicts a 2D projection of the outer boundary of the object inthe pose as shown in FIG. 13C.

FIG. 15 depicts a comparison of a 2D perimeter contour line of a bonewith a plurality of projections of poses of a bone from a 3D bone model.

FIG. 16 depicts a lateral-to-medial or medial-to-lateral X-ray image ofa foot including an identification of various landmarks therein.

FIG. 17 is a flowchart depicting an exemplary method of constrainingavailable poses for pose estimation using bio-kinematics of the foot.

FIG. 18 is an example computing system having one or more computingunits that may implement various systems and methods discussed herein.

DETAILED DESCRIPTION

Aspects of the present disclosure involve mapping weight bearingconditions of the foot and leg from X-ray images obtained with astanding patient to the 3D bone model of the foot and leg obtained fromCT images of the patient in a non-standing, supine position. Suchmapping is beneficial in aligning the foot along its natural or weightedposition with respect to the tibia.

And while the disclosure describes mapping weight bearing conditions ofthe foot from standing images to 3D bone models of the foot obtainedfrom non-standing images, the disclosure also encompasses mapping weightbearing conditions of other bones and joints of the body including hipsand knees, among other joints and bones making up the joints withoutlimitation. For example, standing X-ray images of a patient's hip regionor knee region may be acquired, as well as non-weight bearing images ofthe patient's hip region or knee region, respectively. 3D bone modelsmay be generated of the patient's hip region or knee region, and thepose of the bones of the 3D bone models may be modified based on theposes of the bones in the standing X-ray images. In the case of mappingweight bearing conditions of an X-ray to a 3D bone model of a patient'ship, the standing and non-standing images may show differentrelationships between the ileum and the femur and tibia. Similarly, inthe case of mapping weight bearing conditions of an X-ray to a 3D bonemodel of a patient's knee, the standing and non-standing images may showdifferent relationships between the femur and tibia.

The following discussion includes three methods of positionallymodifying the 3D bone models generated from CT images based oninformation from weighted pose of the foot and leg bones in the X-rayimages. The three methods are: pose estimation via comparison of 2dimage and 3d bone model in common coordinate system; pose estimation via2d comparison of x-ray and plurality of bone model projections; andaugmented pose estimation.

I. Pose Estimation Via Comparison of 2D Image and 3D Bone Model inCommon Coordinate System

The manual pose estimation method is where the 3D bone models of apatient's leg and foot bones (e.g., tibia, fibula, talus, calcaneus)obtained from segmenting the foot and leg from the CT images areimported into a 3D coordinate system of a computer along with X-rayimages of the same patient's leg and foot. As discussed previously,however, the X-ray images depict a standing pose or orientation of thebones of the foot and leg. In certain instances, the 3D bone model in afirst pose may be overlaid or superimposed on top of the X-ray image,which depict the bones in a different (standing) pose. Since the X-rayimages are lateral views and anteroposterior views, the 3D bone modelsmay be interchangeably shown in lateral views and anteroposterior viewsto match the X-ray images in the same lateral views and anteroposteriorviews, respectively. And the orientation of the individual bones of thebone model may be altered to match the pose of the bones in the standingX-ray images.

To begin, reference is made to FIG. 1A, which is a flowchart listingsteps of an exemplary method 1000 of pose estimation ofthree-dimensional bone models in surgical planning a total anklereplacement procedure. As seen in the figure, step 100 of the method1000 includes generating images (in the case of sequential images) or animage dataset (in the case of a continuous volumetric dataset produced,for example, via a helical CT scan) of a patient's foot in a non-weightbearing condition.

Corresponding to step 100 of FIG. 1A, reference is made to FIG. 2, whichis an overhead view of a patient 100 laying supine (i.e., on his or herback) on an imaging table 102 proximate an image scanning machine 104(e.g., computed tomography (CT), magnetic resonance imaging (MRI),ultrasound) in communication with a computer 106. As seen in the figure,the imaging table 102 may be a motorized platform that is aligned withan opening 108 in a ring 110 of the image scanning machine 104. Theimaging table 102 may be translated so a portion of the table 102supporting the patient 100 extends into the opening 108 of the ring 110of the machine 104. For example, in the case of an imaging procedure onthe patient's leg and foot 112, the table 102 may be translated untilthe patient's leg and foot 112 is within the ring 110 of the machine104. In the case of a CT image scanning machine 104, a scan of thepatient may produce a volumetric dataset or individual slices (e.g.,axial, coronal, sagittal). Most conventional CT image scanning machines104 are helical computed axial tomography machines where the x-ray beamtraces a helical path relative to the patient, as the patient istranslated relative to the machine. The helical CT machines 104 producea volumetric scan that can be reconstructed into sequential images witha defined spacing, which is a similar result to sequential scanningacquisition machines 104 that perform scans at a pre-defined spacing asthe gantry moves the patient sequentially through the ring 110 of themachine 104. Helical CT machines 104 may be advantageous because ahelical scan of the patient can be performed in seconds, whereas asequential scan can take tens of minutes.

As the patient's leg and foot 112 is scanned via the scanning machine104, the computer 116 stores the raw data obtained from the image scan,and may process the data so it is useable by a user (e.g., radiologist,engineer, surgeon). The raw data may be processed and stored as aDigital Imaging and Communications in Medicine (“DICOM”) file. The DICOMfile is a communication protocol and a file format for storing medicalinformation, such as the volumetric dataset of the patient's foot, forexample. Using a DICOM viewer program, the DICOM file can be opened andthe data can be viewed in various forms. For example, volumetric datafrom a helical scan can be reconstructed into various two-dimensional(“2D”) views such as axial, sagittal, and coronal views. The data mayadditionally or alternatively be processed into Digitally ReconstructedRadiographs (“DRR”).

Upon processing of the raw data from the image scanning machine 104 viathe computer 116, exemplary 2D images can be seen in FIG. 6, whichspecifically shows a stack 114 of scan or slice images 116 of thepatient's leg and foot 112. In this example, the images 116 may be 2Dimages reconstructed from a volumetric dataset acquired via a helical CTmachine 104 or from a CT machine 104 that acquired sequential images. Asseen in FIG. 6, the images 116 are sagittal image slices 116 of the legand foot 112. The entire stack 114 of image slices 116 may make up theentirety of the patient's leg and foot 112 from a medial side to alateral side.

The images 116 of the patient's leg and foot 112 are with the patientlying on the imaging table 102 in a supine position. That is, thepatient's leg and foot 112 is unweighted, or in a non-weight bearingcondition. Thus, the images 116 taken with the imaging machine 104 showthe bones of the leg and foot 112 in an uncompressed or non-load bearingfashion.

Step 100 may be described as a computer process that includes a step ofreceiving first patient bone data 116, 114 of a patient leg and foot 112in a first pose. The first patient bone data 116, 114 may be generatedvia a first imaging modality such as CT or MRI. The first pose mayinclude a position and orientation of the patient leg relative to thepatient foot 112 as defined in the first patient bone data 116, 114.

The images 116 of the patient's leg and foot 112 in a non-weight bearingcondition are in contrast to X-ray images of the patient's foot in aweight bearing condition. Referring back to FIG. 1A, step 102 of themethod 1000 may include generating two-dimensional (“2D”) images of thepatient's leg and foot in a weight bearing condition. To that end,reference is made to FIGS. 3 and 4, which depict, respectively, thepatient 100 having a medial and an anteroposterior X-ray of the leg andfoot 112 with the patient 100 in a standing position. As seen in FIG. 3,a generator 118 and a detector 120 of an X-ray machine 122 arepositioned on either side of the foot 112 of the patient 100. Thegenerator 118 is positioned on the medial side of the foot 112, and thedetector 120 is positioned on the lateral side of the foot 112 in FIG.3. Thus, the resulting 2D X-ray image 124 of the foot 112 in a medial orlateral view can be seen in FIG. 5A. In certain instances, a lateral 2DX-ray image of the patient foot 112 may also be generated (not shown).

As seen in FIG. 4, the generator 118 is positioned in front of the foot112 and the detector 120 is positioned behind the foot 112 so as toproduce an anteroposterior (“AP”) X-ray image of the foot 112. Theresulting 2D X-ray image 126 of the foot 112 in the anteroposteriorposition can be seen in FIG. 5B. The images 124, 126 of the patient'sfoot 112 produced via the X-ray machine 122 depict the bones of the footin a weighted condition since the X-rays were performed with the patient100 standing. While X-rays of the foot 112 are described as beingperformed in the medial, lateral, and anteroposterior views, X-rays maybe taken of the foot 112 in additional or alternative views withoutdeparting from the teachings of the present disclosure.

Step 102 may be described as a computer process that includes a step ofreceiving second patient bone data 124, 126 of the patient leg and foot112 in a second pose. The second patient bone data 124, 126 may begenerated via a second imaging modality that is different from the firstimaging modality such as X-ray. The second pose may include a positionand orientation of the patient leg relative to the patient foot 112 asdefined in the second patient bone data 124, 126.

Referring back to the method 1000 in FIG. 1A, step 104 may includesegmenting the bones of the patient's foot in the images of the foot inthe non-weight bearing condition. As seen in FIG. 6, the individualbones (tibia 128, talus 130, calcaneus 132, navicular 134, cuneiforms136, metatarsals, phalanges, fibula, etc.) of the leg and foot 112 maysegmented along their respective bone boundaries in each of the images116 of the stack 114. While FIG. 6 illustrates the calcaneus 132 and thetalus 130 segmented along their respective bone boundaries (i.e., thewhite line around the perimeter of the bones) in a single image 116, allbones or only the bones relevant to the TAR procedure may be segmentedin all of the images 116 of the stack 114. Thus, after the segmentationprocess of step 104 of the method 1000 of FIG. 1A, all images 116 of thestack 114 may include the bones of the foot 112 segmented along theirrespective bone boundaries.

Next, step 106 of the method 1000 of FIG. 1A may include generating athree-dimensional (“3D”) bone model of the patient's foot 112 from thesegmented images 116 of the stack 116. An exemplary bone model 138 ofthe patient's foot 112 formed from the individually segmented bones inthe images 116 of the stack 116, and displayed on a display screen 140of a display device 142, may be seen in FIG. 7. Generation of the bonemodel 138 may be done via the computer 106 by interpolating a surfacemesh between the spaces between the individually segmented images 116 toform a 3D surface profile approximating the surface contours of thebones of the patient's foot. As seen in FIG. 7, the bones of the foot112 and the leg, including the tibia and fibula are at least partiallygenerated into 3D form.

Exemplary computer programs for generating the 3D bone model 138 fromthe images 116 may include: Analyze from AnalyzeDirect, Inc., OverlandPark, Kans.; Insight Toolkit, an open-source software available from theNational Library of Medicine Insight Segmentation and RegistrationToolkit (“ITK”), www.itk.org; 3D Slicer, an open-source softwareavailable from www.slicer.org; Mimics from Materialise, Ann Arbor,Mich.; and Paraview available at www.paraview.org, among others.

Step 106 may be described as a computer process including a step ofgenerating a three-dimensional (3D) bone model 138 of the patient legand foot 112 from the first patient bone data 114, 116, where the 3Dbone model includes a plurality of 3D bone models arranged in the firstpose.

Referring back to FIG. 1A, step 108 of the method 1000 may includeimporting the 3D bone model 138 of FIG. 7 and the 2D images of FIGS. 5Aand 5B into a common coordinate system. FIG. 8 illustrates the 3D bonemodel 138 and the 2D image 124 (medial or lateral view) of the patient'sfoot 112 in a common coordinate system (x,y,z). Since the 2D image 124is a medial or lateral view in this example (planar views of the bonesof the foot 112), the 3D bone model 138 may also be oriented in the samemedial or lateral view. As seen in FIG. 8, the 3D bone model 138 isoriented in a lateral view to match a lateral 2D image 124.

Referring to FIG. 1B, which is a continuation of the method 1000 of FIG.1A, step 110 may include orienting the 3D bone model 138 in a matchingorientation with one or both of the 2D images 124, 126. For example, asseen in FIG. 8, the 3D bone model 138 is oriented in a lateral view thatmatches the lateral 2D X-ray image 124. The 3D bone model 138 and the 2Dimage 124 are displayed on the display screen 140 of the display device142 (e.g., computer 106, tablet). The 2D image 126 of the foot 112 inthe anteroposterior view, of FIG. 5B, may not be shown in FIG. 8 becausethe image 126, which lies in a plane (y, z plane), is perpendicular tothe plane (x, y plane) of the image 124 in FIG. 8. But, the orientationof the bone model 138 may be changed to the y, z plane such that the 2DX-ray image 126 of the foot 112 in the anteroposterior view is shown.

Step 112 of the method 1000 of FIG. 1B may also include scaling at leastone of the 3D bone model 138 and the 2D images until the scales of thebones are the same. Since the 2D images 124, 126 are already the samesize, the bone model 138 may be scaled to match the size of the bones inthe 2D images 124, 126 with a single step. Alternatively, the scale ofthe 2D images 124, 126 individually or in combination may be scaled tomatch the size of the 3D bone model 138. The scaling may be performedmanually or automatically. The scale of the 3D bone model 138 and the 2Dimage 124 are the same in FIG. 8.

Step 114 of the method 1000 of FIG. 1B may also include orienting theindividual bones of the 3D bone model 138 to match the orientation ofthe individual bones of in the 2D images 124, 126. This step may includestep 116, which may include translating and/or rotating the individualbones of the 3D bone model 138 in medial, lateral, and anteroposteriorviews to match the orientation of the bones in the corresponding 2Dimages 124, 126. This step may be described as a computer processincluding a step of modifying the 3D bone model 138 of the patient legand foot 112 such that the plurality of 3D bone models are reorientedinto a third pose that matches a particular arrangement of bones in thepatient leg and foot in the second pose. In certain instances, the stepof modifying the 3D bone model 138 may be manual, automatic, orpartially manual and partially automatic.

As seen in FIG. 9, which is a view of the 2D image 124 of the foot 112in the weighted condition overlaid with the 3D bone model 138 of thefoot in the non-weighted condition, the bones of the foot in the bonemodel 138 have been rotated clockwise and translated in the y-directionuntil a proximal surface of the talus in the bone model 138 iscoextensive or overlaps with the proximal surface of the talus in the 2DX-ray image 124. As seen in FIG. 9, the individual bones of the 3D bonemodel 138 have not yet been moved relative to each other. Instead, theentire set of bones forming the 3D bone model 138 have been roughlyaligned with the bones of the foot 112 in the X-ray image 124.

Reference is made to FIG. 10, which is the same views of the 3D bonemodel 138 and 2D X-ray image 124 displayed on the display screen 140 ofthe display device 142 of FIG. 9, except the individual bones of the 3Dbone model 138 have been translated and/or rotated relative to eachother so as to match the pose of the bones of the foot 112 in the X-rayimage. As seen in FIG. 10, an outline or projection of the bones of thebone model 138 have been adjusted or reoriented relative to each otherso as to match the positioning/spacing orientation of the bones in the2D X-ray image 124 in the medial view. The same process may take placefor adjusting the orientation of the bones in the y, z plane withrespect to the 2D X-ray image 126 of the foot 112 in the anteroposteriorview, as well as other views including but not limited to the medialview, dorsal view, etc.

The 3D bone model 138 of the foot 112, in FIGS. 9 and 10, may beencircled by a rotation tool (not shown) indicating the computer 106 mayrotate the selected 3D bone model of the foot within the particularplane to match the particular pose of the foot 112 in the X-ray image124. The rotation tool may rotate the 3D bone models 138 about any axis(e.g., x,y,z) to align the models with the X-ray images. The GUI of thecomputer 106 may include a translation tool (not shown) for translatingany of the bones of the bone model 138 in a particular direction (e.g.,x, y, z). Particularly, the GUI may permit the translation tool 146 tomove the bones of the bone model 138 in an x-direction or y-direction.On the GUI of the computer 106, there may be a selection drop down forswitching each of the 3D bone models 138 between cuneiform, cuboid,metatarsals, calcaneus etc. Additionally, the GUI of the computer mayalso allow the switching between rotation, scale and translation modes.

The 3D bone models 138 and X-ray images 124, 126 may be iterativelytranslated, rotated, and/or scaled till the bone contour lines (outermost boundary as projected on a plane) align with each other.Additionally or alternatively, certain bone landmarks on the bonesurface may be identified in each of the 3D bone models 138 and X-rayimages 124, 126 and the landmarks may be positionally matched such thatthey align with each other. Instead of surface landmarks, a centroid ofthe 3D bone models may be identified and similarly identified in thelateral and anteroposterior views of the X-rays 124, 126, and thecentroids can be matched so the models and X-rays align with each other.

In certain instances, accuracy of mapping the 3D bone models to theX-ray images may be improved by introducing pick able landmarks on theX-ray and the Bone mesh for correspondence.

II. Pose Estimation Via 2D Comparison of X-Ray and Plurality of BoneModel Projections

In the automated pose estimation, the mapping of the 3D bone models tothe X-ray images may be fully or partially automated. One such method2000 for automated pose estimation of a 3D bone model 138 may be seen inthe flowchart of FIGS. 11A-11C. Referring to FIG. 11A, the method 2000may include steps 200, 202, and 204, which are identical to steps 100,104, and 106 of the method 1000 described in reference to FIG. 1A, amongothers. Thus, steps 200, 202, and 204 will not be described in detail;instead, please refer to the previous discussion of steps 100, 104, and106 for a detailed description. Generally, step 200 may includegenerating images or an image dataset (e.g., CT images, MRI, ultrasoundimages) of the patient's foot 112 in a non-weight bearing condition suchas, for example, with the patient 100 laying supine on a an imagingtable 102. At step 202, the bones of the foot 112 as seen in the imagesor image dataset may be segmented (e.g., along the bone contour lines insagittal, axial, or coronal images). At step 204, a 3D bone model 138 ofthe patient's foot 112 may be generated from the segmented images.

Step 210 of FIG. 11A is the same as step 102 of FIG. 1A; therefore, adetailed discussion of this step will not be included for the method2000 in FIG. 11A. Please refer to the details of step 102 regarding thespecifics of step 210. Generally, step 210 may include generating 2DX-ray images 200 (as seen in the lateral X-ray view of FIG. 12A and themedial X-ray view of FIG. 12B) of the patient's foot 112 in a weightbearing condition (e.g., standing X-rays). Medial, lateral, andanteroposterior views (seen in FIG. 5B), among others, may be generated.The X-ray images 200 of FIGS. 12A and 12B are illustrative and may belabeled as: right foot, lateral-to-medial view, and right footmedial-to-lateral view; or left foot, medial-to-lateral view, and leftfoot lateral-to-medial view, respectively, as it can be difficult orimpossible to determine the views of X-rays without labeling.

Step 212 of FIG. 11A may include segmenting the individual bones of thefoot 112 in the 2D X-ray images 200. As seen in FIG. 12A, the talus 202and the calcaneus 204 are segmented from the 2D X-ray image 200 in thelateral view, and the same bones are segmented from the 2D X-ray image200 in the medial view of FIG. 12B. Segmentation may also take placewith a 2D X-ray image 200 in an anteroposterior view (not shown), amongothers.

In FIGS. 12A and 12B, only the talus 202 and calcaneus 204 are shownsegmented, but other bones of the foot 112 including the tibia,navicular, cuneiforms, metatarsals, and phalanges, among others, may besegmented. The bones of the talus 202 and calcaneus 204 are exemplaryillustrations, but the method 2000 is intended to include thesegmentation or additional or alternative bones of the foot 112depending on the bones desired to be estimated in their pose.

Following segmentation in step 212, step 214 of FIG. 11A may includegenerating a bone contour line along the perimeter of each of thesegmented bones of the foot 112. As seen in FIGS. 12A and 12B, beneaththe segmented talus 202 is a contour line 206 defining a perimeter ofthe segmented talus 202, and beneath the segmented calcaneus 204 is acontour line 208 defining a perimeter of the segmented calcaneus 204.The perimeter contour lines 206, 208 represent an outer shape of thebones of the talus 202 and calcaneus 204 in their particular pose(position and orientation) when standing in the X-ray image 200.

Turning back to the method 2000 as seen in FIG. 11B, and continuing fromstep 204, step 206 may include generating a plurality of poses of eachof the individual bones making up the 3D bone model 138. FIGS. 13A, 13B,and 13C illustrate a 3D object 210 such as the individual bones makingup the 3D bone model 138 of the foot in various poses, each having adifferent pose. FIG. 13A shows the object 210 a in a first pose, FIG.13B shows the object 210 b in a second pose, and FIG. 13C shows theobject 210 c in a third pose. As described herein, pose refers to theposition and orientation of an object in space. Therefore, it can beseen in FIGS. 13A, 13B, and 13C that the 3D objects 210 a, 210 b, 210 care in different rotation orientations relative to each other, eachhaving been rotated along various axes. The 3D objects 210 a, 210 b, 210c may represent the individual bones of the 3D bone model 138 havingbeen generated in a plurality of poses. Since there are many bones ofthe foot 112, and there are a near infinite number of poses for each ofthe bones of the foot 112, it is most efficient to describe the objects210 a, 210 b, 210 c as representing the plurality of poses of the bonesof the foot 112.

In certain instances, a certain number of finite poses of each of thebones of the foot 112 may be generated. In certain instances, onehundred different poses of each of the bones of the foot 112 may begenerated. In certain instances, five hundred different poses of each ofthe bones of the foot 112 may be generated. In certain instances, onethousand different poses of each of the bones of the foot 112 may begenerated. In certain instances, the poses of each of the bones of thefoot 112 can be changed in any one or multiple of the six degrees offreedom (three translations and three rotations). The smaller thedifferences among the poses (e.g., a change of 1 degree of rotation onan axis for each different pose), the higher the number of poses thatwill be generated. In contrast, the larger the differences between theposes (e.g., a change of 10 degrees of rotation on an axis for eachdifferent pose), the fewer the number of poses that will be generated.

Step 216 of FIG. 11B may include generating a plurality of projectionsfrom the plurality of poses of each of the bones of the foot 112. Asseen in FIGS. 14A, 14B, and 14C, a projection (i.e., in a plane, in 2D)or outline of the perimeter 212 of each of the plurality of poses of thebones (or object 210 as seen in FIGS. 13A, 13B, and 13C) is generated.As seen in FIGS. 14A, 14B, and 14C, the projections 212 a, 212 b, 212 cof the poses of the objects 210 a, 210 b, 210 c are different for eachpose, where the first pose of the object 210 a in FIG. 13A yields theprojection 212 a in FIG. 14A. Similarly, the second pose of the object210 b in FIG. 13B yields the projection 212 b in FIG. 14B, and the thirdpose of the object 210 c in FIG. 13C yields the projection 212 c in FIG.14C.

Referring back to the method 2000 of FIG. 11B, step 216 may include acomparison step that includes comparing, for each bone of the foot 112of interest, the bone contour line as determined from the 2D X-rayimages 200 (contour line 206 for the talus, and contour line 208 for thecalcaneus in FIGS. 12A and 12B) to the plurality of projections(projections 212 a, 212 b, and 212 c in FIGS. 14A-14C) as determinedfrom the 3D bone model 138, and identifying a particular projection fromthe plurality of projections 212 a, 212 b, 212 c that best-fits or mostclosely matches the bone contour line.

FIG. 15 illustrates step 216 of the method 2000. As seen in the figure,the talus contour lines 206 as identified from the 2D X-ray images 200is compared to an example plurality of projections 212 d, 212 e, 212 f,and 212 g, and a particular one of the plurality of projections 212 g isidentified as being the best-fit or closest match in shape andorientation. This comparison step 216 may include a shape matchingalgorithm that compares the shape and area within its boundary of eachof the plurality of projections 212 d-g to the shape and area within theboundary of the talus contour line 206. The particular projection 212 gof the plurality of projections 212 d-g that most closely matches thevalues of the talus contour line 206 is identified as the best-fit ormost closely matching.

Each of the plurality of projections 212 may be sampled radially in theform of a shape context. And the data for each of the plurality ofprojections 212 may be compared with the shape context of the contourline 206.

The comparison and identification step 216 may include employing aJaccard similarity coefficient for comparing the similarity anddiversity contour line as determined from the 2D X-ray image to each ofthe plurality of projections as determined from the 3D bone model 138.In comparing the contour line 206 to each of the projections 212 d-g, asseen in FIG. 15, a Jaccard similarity coefficient may be assigned toeach comparison. The assigned coefficient can be used to determine whichpair is the most similar. An example Jaccard similarity measurement maybe defined as: Jaccard Similarity J (A, B)−I Intersection (A, B) I/IUnion (A, B) I.

An example shape matching algorithm that may be employed in the method2000 for comparing and identifying the particular projection that mostclosely matches the contour line as determined from the 2D X-ray images200 may be seen in the following document, which is hereby incorporatedby reference in its entirety: “A Comparison of Shape Matching Methodsfor Contour Based Pose Estimation” by Bodo Rosenhahn, Thomas Brox,Daniel Cremers, and Hans-Peter Seidel(https://vision.in.tum.de/_media/spezial/bib/rosenhahn_iwcia06.pdf).

Step 216 may be employed for each individual bone of interest. That is,while FIG. 15 only depicts the talus, step 216 may be employed foradditional or alternative bones including the tibia, calcaneus,navicular, cuneiforms, metatarsals, phalanges, etc.

Referring back to FIG. 11B, the method 2000 at step 218 may includeerror checking via an iterative process to determine if there are anyadditional poses that provide a better-fit than the particular poseidentified from the plurality of poses. Step 218 may include modifyingthe plurality of poses 210 and projections 212, and running thecomparison and identification step of step 216 again to see if there arebetter-fit poses.

Once the iterative process has ran and the best-fitting projections havebeen determined, the individual bones of the bone model 138 may bearranged according to the orientation of the identified particularprojection for each of the bones, and the individual bones of the bonemodel 138 may be arranged relative to each other according to theirspacing in the 2D X-ray images 200, as seen in step 220 of FIG. 11C.

Stated differently, each of the particular projections identified asbeing the best fit to the contour lines determined from the 2D X-rayimages 200 determines the orientation of the individual bones of thebone model 138. Step 220 may include arranging the individual bones ofthe bone model 138 according to their respective particular projectionthat was identified as the best fit with the contour lines determinedfrom the 2D X-ray images 200.

Once the bones of the bone model 138 are arranged according to step 220,the bone model 138 is in a pose that matches or replicates a weightedcondition of the foot 112 as it appeared in the 2D X-ray images 200.

Referring to FIG. 11C, the method 2000 may additionally includepreoperatively planning the TAR procedure, at step 222. This step mayinclude determining resection placement, resection depth, implantplacement, implant depth, implant type and size, and surgical approach,among other parameters.

Step 224 of the method 2000 may then include performing the TARprocedure according to the preoperative plan at step 222. This mayinclude sedating the patient, creating an incision into the patient'sskin, resecting bone, implanting a fixation device or implant, andclosing the incision, among other steps of a TAR procedure.

Referring back to FIG. 11C, the method 2000 may include, at step 226,applying bio-kinematic constraints to the generation of the plurality ofposes for each of the bones making up the 3D bone model 138. Thebio-kinematic constraints may limit the number of poses generated or thenumber of projections that are ultimately generated at step 208. Thebio-kinematic constraints may limit the poses to those that arebio-kinematically relevant, whereas such poses that are notbio-kinematically relevant will not be generated, or will be discarded.

In certain instances, the bio-kinematic constraints may includeorientation guidelines for each bone as it relates to surrounding bonesgiven a known view (e.g., lateral, medial, anteroposterior). That is, asseen in FIG. 16, which is a lateral-to-medial or medial-to-lateral X-Rayimage 200 of the right foot 112, the talus 214, for example, includes aconvex head 216 that articulates with a concave articular surfaceportion 218 on the superior surface of the navicular 220. The talus 214also includes a large posterior facet 222 abutting the calcaneus 224,and a large superior articular surface (talar dome) 226 for abutting theinferior aspect of the tibia 228. Thus, if the view is known (e.g.,lateral, medial, AP), certain constraints can be built into the systemthat exclude poses that are not relevant. For example, in a lateral viewof the foot as seen in FIG. 16, certain parts of the bones may beidentified in relevant bones. For the calcaneus 224, the calcanealtuberosity 230, the posterior facet 232, and facet for cuboid 234 may beidentified in the X-Ray image 200 or the segmented image of thecalcaneus (not shown). For the talus 214, the talar dome 226, and head216 may be identified in the X-Ray image 200 or the segmented image ofthe talus (not shown). For the navicular 220, the superior articularsurface 218 for abutting the talus head 216 may be identified in theX-Ray image 200 or the segmented image of the navicular (not shown).

Once these points are identified on the relevant bones, certain posescan be eliminated that do not meet the bio-kinematics of the foot. Forinstance, the calcaneal tuberosity 230 must be at a left-most positionin a lateral view of the right foot. The calcaneal posterior facet 232generally faces oppositely of the calcaneal tuberosity 230, and abutsthe talus 214. The calcaneal facet for the cuboid 234 is generally in afar right position in the lateral view of the right foot. For the talus214, the talar dome 226 is generally oriented upwards, facing the distaltibia 228. And the talar head 216 generally faces to the right in thelateral view of the right foot. For the navicular 220, the superiorarticular surface 218 generally faces to the left in the lateral view ofthe right foot.

All this information can be used to constrain the poses generated atstep 206 by eliminating poses that have, for example: the calcanealtuberosity 230 at a far right position in a lateral view of the rightfoot 112; the calcaneal facet for the cuboid 234 that faces left; talardome 226 facing downward or to the right; talar head 216 facing left;and superior articular surface 218 of the navicular 220 facing right;among others.

In certain instances, the 3D bone model 138 may be modeled usinglandmarks. For instance, the articular surfaces of the bones may beidentified and the poses from step 206 may be limited to orientationsthat require the articular surfaces to oppose each other and be acertain distance from each other. Certain motion of the joints may alsobe used as constraints. For instance, the forefoot may be modeled as ahinge joint, and the talocrural joint can be modeled as a hinge jointwith rotation axis about the line on the superior point of lateral andmedial malleolus. Thus, certain poses that do not permit such rotationabout the rotation axis may be eliminated.

FIG. 17 is a flowchart depicting an exemplary method 3000 forconstraining the poses of step 206 using bio-kinematics (step 226). Themethod 3000 may include, at step 228, identifying landmarks in each ofthe individual bones of the 3D bone model 138. This step may includeperforming a topological data analysis (“TDA”) to extract informationfrom the dataset associated with each bone of the bone model 138. Next,at step 230, the method 3000 may include building coordinate frames foreach of the bones in the bone model 138 from the identified landmarks ofstep 228. At step 232, the method 3000 may further include identifyingpermissible relationships (e.g., rotational orientation, translationalorientation) between the coordinate frames for each bone of the bonemodel 138. At step 234, the method 3000 may include providing aplurality of orientations of the bones within permissible rotational andtranslational relationships for the generation of poses at step 206.

III. Augmented Pose Estimation

Another method of mapping the 3D bone models 138 to the X-ray images mayinclude an augmented pose estimation, which may be a combination ofmanual and automated procedures. For instance, instead of running acontour matching algorithm as described in Section II. on a complete setof the bones of the leg and foot, the contour matching algorithm may belimited to certain bone structures, such as the fibula, tibia, talus,and, calcaneus. The remaining bones of the foot may be extrapolated fromthe resulting pose of the fibula, tibia, talus and calcaneus.

In certain instances, a user may manually map bone contour surfaces orlandmarks on the individual bones of the 3D bone model 138 tocorresponding points on the X-ray images, as described in Section I.Then, the user may perform an automation step (as in Section II.) tooptimize the pose further on particular bone structures. In this way,the user provides a “rough” estimate of pose, and the automation processfine-tunes the original “rough” estimate of pose.

In sum, the above described techniques may be used to estimate absolutepose of a foot 112 in anteroposterior, lateral, and medial views withrespect to tibia and relative bone positions in the foot 112. Thesemethods may improve accuracy in deformity assessment and hencecorrection for TAR procedures.

IV. Example Computing System

Referring to FIG. 18, a detailed description of an example computingsystem 1800 having one or more computing units that may implementvarious systems and methods discussed herein is provided. The computingsystem 1800 may be applicable to any of the computers or systemsutilized in the planning of the TAR procedure, and other computing ornetwork devices. It will be appreciated that specific implementations ofthese devices may be of differing possible specific computingarchitectures not all of which are specifically discussed herein butwill be understood by those of ordinary skill in the art.

The computer system 1800 may be a computing system that is capable ofexecuting a computer program product to execute a computer process. Dataand program files may be input to the computer system 1800, which readsthe files and executes the programs therein. Some of the elements of thecomputer system 1800 are shown in FIG. 18, including one or morehardware processors 1802, one or more data storage devices 1804, one ormore memory devices 1808, and/or one or more ports 1808-1810.Additionally, other elements that will be recognized by those skilled inthe art may be included in the computing system 1800 but are notexplicitly depicted in FIG. 18 or discussed further herein. Variouselements of the computer system 1800 may communicate with one another byway of one or more communication buses, point-to-point communicationpaths, or other communication means not explicitly depicted in FIG. 18.

The processor 1802 may include, for example, a central processing unit(CPU), a microprocessor, a microcontroller, a digital signal processor(DSP), and/or one or more internal levels of cache. There may be one ormore processors 1802, such that the processor 1802 comprises a singlecentral-processing unit, or a plurality of processing units capable ofexecuting instructions and performing operations in parallel with eachother, commonly referred to as a parallel processing environment.

The computer system 1800 may be a conventional computer, a distributedcomputer, or any other type of computer, such as one or more externalcomputers made available via a cloud computing architecture. Thepresently described technology is optionally implemented in softwarestored on the data stored device(s) 1804, stored on the memory device(s)1806, and/or communicated via one or more of the ports 1808-1810,thereby transforming the computer system 1800 in FIG. 18 to a specialpurpose machine for implementing the operations described herein.Examples of the computer system 1800 include personal computers,terminals, workstations, mobile phones, tablets, laptops, personalcomputers, multimedia consoles, gaming consoles, set top boxes, and thelike.

The one or more data storage devices 1804 may include any non-volatiledata storage device capable of storing data generated or employed withinthe computing system 1800, such as computer executable instructions forperforming a computer process, which may include instructions of bothapplication programs and an operating system (OS) that manages thevarious components of the computing system 1800. The data storagedevices 1804 may include, without limitation, magnetic disk drives,optical disk drives, solid state drives (SSDs), flash drives, and thelike. The data storage devices 1804 may include removable data storagemedia, non-removable data storage media, and/or external storage devicesmade available via a wired or wireless network architecture with suchcomputer program products, including one or more database managementproducts, web server products, application server products, and/or otheradditional software components. Examples of removable data storage mediainclude Compact Disc Read-Only Memory (CD-ROM), Digital Versatile DiscRead-Only Memory (DVD-ROM), magneto-optical disks, flash drives, and thelike. Examples of non-removable data storage media include internalmagnetic hard disks, SSDs, and the like. The one or more memory devices1806 may include volatile memory (e.g., dynamic random access memory(DRAM), static random access memory (SRAM), etc.) and/or non-volatilememory (e.g., read-only memory (ROM), flash memory, etc.).

Computer program products containing mechanisms to effectuate thesystems and methods in accordance with the presently describedtechnology may reside in the data storage devices 1804 and/or the memorydevices 1806, which may be referred to as machine-readable media. Itwill be appreciated that machine-readable media may include any tangiblenon-transitory medium that is capable of storing or encodinginstructions to perform any one or more of the operations of the presentdisclosure for execution by a machine or that is capable of storing orencoding data structures and/or modules utilized by or associated withsuch instructions. Machine-readable media may include a single medium ormultiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) that store the one or more executableinstructions or data structures.

In some implementations, the computer system 1800 includes one or moreports, such as an input/output (I/O) port 1808 and a communication port1810, for communicating with other computing, network, or other devices.It will be appreciated that the ports 1808-1810 may be combined orseparate and that more or fewer ports may be included in the computersystem 1800.

The I/O port 1808 may be connected to an I/O device, or other device, bywhich information is input to or output from the computing system 1800.Such I/O devices may include, without limitation, one or more inputdevices, output devices, and/or environment transducer devices.

In one implementation, the input devices convert a human-generatedsignal, such as, human voice, physical movement, physical touch orpressure, and/or the like, into electrical signals as input data intothe computing system 1800 via the I/O port 1808. Similarly, the outputdevices may convert electrical signals received from computing system1800 via the I/O port 1808 into signals that may be sensed as output bya human, such as sound, light, and/or touch. The input device may be analphanumeric input device, including alphanumeric and other keys forcommunicating information and/or command selections to the processor1802 via the I/O port 1808. The input device may be another type of userinput device including, but not limited to: direction and selectioncontrol devices, such as a mouse, a trackball, cursor direction keys, ajoystick, and/or a wheel; one or more sensors, such as a camera, amicrophone, a positional sensor, an orientation sensor, a gravitationalsensor, an inertial sensor, and/or an accelerometer; and/or atouch-sensitive display screen (“touchscreen”). The output devices mayinclude, without limitation, a display, a touchscreen, a speaker, atactile and/or haptic output device, and/or the like. In someimplementations, the input device and the output device may be the samedevice, for example, in the case of a touchscreen.

In one implementation, a communication port 1810 is connected to anetwork by way of which the computer system 1800 may receive networkdata useful in executing the methods and systems set out herein as wellas transmitting information and network configuration changes determinedthereby. Stated differently, the communication port 1810 connects thecomputer system 1800 to one or more communication interface devicesconfigured to transmit and/or receive information between the computingsystem 1800 and other devices by way of one or more wired or wirelesscommunication networks or connections. Examples of such networks orconnections include, without limitation, Universal Serial Bus (USB),Ethernet, Wi-Fi, Bluetooth®, Near Field Communication (NFC), Long-TermEvolution (LTE), and so on. One or more such communication interfacedevices may be utilized via the communication port 1810 to communicateone or more other machines, either directly over a point-to-pointcommunication path, over a wide area network (WAN) (e.g., the Internet),over a local area network (LAN), over a cellular (e.g., third generation(3G) or fourth generation (4G)) network, or over another communicationmeans. Further, the communication port 1810 may communicate with anantenna or other link for electromagnetic signal transmission and/orreception.

In an example implementation, patient data, bone models, transformation,mapping and shape matching software, tracking and navigation software,registration software, and other software and other modules and servicesmay be embodied by instructions stored on the data storage devices 1804and/or the memory devices 1806 and executed by the processor 1802. Thecomputer system 1800 may be integrated with or otherwise form part of asurgical system for planning and performing a TAR procedure.

The system set forth in FIG. 18 is but one possible example of acomputer system that may employ or be configured in accordance withaspects of the present disclosure. It will be appreciated that othernon-transitory tangible computer-readable storage media storingcomputer-executable instructions for implementing the presentlydisclosed technology on a computing system may be utilized.

In the present disclosure, the methods disclosed herein may beimplemented as sets of instructions or software readable by a device.Further, it is understood that the specific order or hierarchy of stepsin the methods disclosed are instances of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of steps in the method can be rearranged while remainingwithin the disclosed subject matter. The accompanying method claimspresent elements of the various steps in a sample order, and are notnecessarily meant to be limited to the specific order or hierarchypresented.

The described disclosure including any of the methods described hereinmay be provided as a computer program product, or software, that mayinclude a non-transitory machine-readable medium having stored thereoninstructions, which may be used to program a computer system (or otherelectronic devices) to perform a process according to the presentdisclosure. A machine-readable medium includes any mechanism for storinginformation in a form (e.g., software, processing application) readableby a machine (e.g., a computer). The machine-readable medium mayinclude, but is not limited to, magnetic storage medium, optical storagemedium; magneto-optical storage medium, read only memory (ROM); randomaccess memory (RAM); erasable programmable memory (e.g., EPROM andEEPROM); flash memory; or other types of medium suitable for storingelectronic instructions.

An example system for processing patient data so as to map weightbearing considerations from standing X-ray images to bones of a 3D bonemodel may include the following components: a network interfaceconfigured to receive one or more sets of patient data; a processingdevice in communication with the network interface; and acomputer-readable medium in communication with the processing deviceconfigured to store information and instructions that, when executed bythe processing device, performs the operations of: receiving firstpatient data 124, 126 comprising at least one two-dimensional (2D) image124, 126 of a patient leg and foot 112 in a weighted pose. Additionaloperations may include receiving second patient data 114, 116 comprisingcomputed tomography (CT) images 114, 116 of the patient leg and foot 112in a non-weighted pose, where the first patient data 124, 126 and thesecond patient data 114, 116 are the result of separate imaging events.Additional operations may include generating a three-dimensional (3D)bone model 138 of the patient leg and foot 112 from the CT images 114,116, where the 3D bone model 138 may include a plurality of 3D bonemodels representing individual bones of the patient leg and foot 112.Additional operations may include rearranging the plurality of 3D bonemodels 138 to mimic the weighted pose of the patient leg and foot 112 inthe at least one 2D image 124, 126.

In certain instances, additional operations may include: generating aplurality of 2D projections of poses 212 of the plurality of 3D bonemodels 138; comparing the plurality of 2D projections 212 to contourlines 206, 208 outlining perimeters of bones of the patient leg and foot112 in the at least one 2D image 124, 128; and identifying particular 2Dprojections 212 g from the plurality of 2D projections 212 that best-fita shape and size of the contour lines 206, 208.

While the present disclosure has been described with reference tovarious implementations, it will be understood that theseimplementations are illustrative and that the scope of the presentdisclosure is not limited to them. Many variations, modifications,additions, and improvements are possible. More generally, embodiments inaccordance with the present disclosure have been described in thecontext of particular implementations. Functionality may be separated orcombined in blocks differently in various embodiments of the disclosureor described with different terminology. These and other variations,modifications, additions, and improvements may fall within the scope ofthe disclosure as defined in the claims that follow.

In general, while the embodiments described herein have been describedwith reference to particular embodiments, modifications can be madethereto without departing from the spirit and scope of the disclosure.Note also that the term “including” as used herein is intended to beinclusive, i.e. “including but not limited to.”

What is claimed is:
 1. One or more tangible computer-readable storagemedia storing computer-executable instructions for performing a computerprocess on a computing system, the computer process comprising:receiving first patient bone data of a patient leg and foot in a firstpose, the first patient bone data generated via a first imagingmodality, the first pose comprising a position and orientation of thepatient leg relative to the patient foot as defined in the first patientbone data; receiving second patient bone data of the patient leg andfoot in a second pose, the second patient bone data generated via asecond imaging modality that is different from the first imagingmodality, the second imaging modality comprising X-ray, the secondpatient bone data comprising X-ray images, the second pose comprising aposition and orientation of the patient leg relative to the patient footas defined in the second patient bone data; segmenting bones of thepatient leg and foot in the X-ray images; generating bone contour linesalong a perimeter of at least some of the bones in the X-ray images;generating a three-dimensional (3D) patient bone model of the patientleg and foot from the first patient bone data, the 3D patient bone modelcomprising a plurality of 3D bone models arranged in the first pose;generating a plurality of two-dimensional (2D) projections from aplurality of poses for each of the plurality of 3D bone models, whereinthe plurality of poses are limited to only such poses that arepermissible given bio-kinematics of bones making up the plurality of 3Dbone models; comparing the bone contour lines to the plurality of 2Dprojections, and identifying particular 2D projections from theplurality of 2D projections that most closely match the bone contourlines; and arranging the plurality of 3D bone models according toorientations represented by the particular 2D projections that wereidentified, and arranging the plurality of 3D bone models relative toeach other according to bone spacing in the second pose.
 2. The one ormore tangible computer-readable storage media of claim 1, wherein thefirst imaging modality is computed tomography or magnetic resonanceimaging.
 3. The one or more tangible computer-readable storage media ofclaim 1, wherein the first pose comprises a non-standing position andorientation of the patient leg relative to the patient foot.
 4. The oneor more tangible computer-readable storage media of claim 1, wherein thesecond pose comprises a standing position and orientation of the patientleg relative to the patient foot.
 5. The one or more tangiblecomputer-readable storage media of claim 1, further comprising importingthe 3D patient bone model and the second patient bone data into a commoncoordinate system.
 6. The one or more tangible computer-readable storagemedia of claim 1, wherein the first patient bone data and the secondpatient bone data are the results of two different imaging events. 7.The one or more tangible computer-readable storage media of claim 1,further comprising: identifying landmarks in the 3D patient bone model;building coordinate frames for the plurality of 3D bone models of the 3Dpatient bone model from the identified landmarks; identifyingpermissible rotational and translational orientations between thecoordinate frames of the plurality of 3D bone models; and providing aplurality of orientations for the plurality of bone models withinpermissible rotational and translational orientations for which to limitthe generation of poses given the bio-kinematics of the bones making upthe plurality of 3D bone models.
 8. One or more tangiblecomputer-readable storage media storing computer-executable instructionsfor performing a computer process on a computing system, the computerprocess comprising: receiving first patient bone data of a patient legand foot in a first pose, the first patient bone data generated via afirst imaging modality, the first pose comprising a position andorientation of the patient leg relative to the patient foot as definedin the first patient bone data; receiving second patient bone data ofthe patient leg and foot in a second pose, the second patient bone datagenerated via a second imaging modality that is different from the firstimaging modality, the second imaging modality comprising X-ray, thesecond patient bone data comprising X-ray images, the second posecomprising a position and orientation of the patient leg relative to thepatient foot as defined in the second patient bone data; segmentingbones of the patient leg and foot in the X-ray images; generating bonecontour lines along a perimeter of at least some of the bones in theX-ray images; generating a three-dimensional (3D) patient bone model ofthe patient leg and foot from the first patient bone data, the 3Dpatient bone model comprising a plurality of 3D bone models arranged inthe first pose; generating a plurality of two-dimensional (2D)projections from a plurality of poses for each of the plurality of 3Dbone models; comparing the bone contour lines to the plurality of 2Dprojections, and identifying particular 2D projections from theplurality of 2D projections that most closely match the bone contourlines; arranging the plurality of 3D bone models according toorientations represented by the particular 2D projections that wereidentified, and arranging the plurality of 3D bone models relative toeach other according to bone spacing in the second pose; andpreoperatively planning a total ankle replacement procedure using theplurality of 3D bone models after being arranged relative to each otheraccording to bone spacing in the second pose.
 9. The one or moretangible computer-related storage media of claim 8, wherein theplurality of poses are limited to only such poses that are permissiblegiven bio-kinematics of bones making up the plurality of 3D bone models.10. The one or more tangible computer-related storage media of claim 9,further comprising: identifying landmarks in the 3D patient bone model;building coordinate frames for the plurality of 3D bone models of the 3Dpatient bone model from the identified landmarks; identifyingpermissible rotational and translational orientations between thecoordinate frames of the plurality of 3D bone models; and providing aplurality of orientations for the plurality of bone models withinpermissible rotational and translational orientations for which to limitthe generation of poses given the bio-kinematics of the bones making upthe plurality of 3D bone models.
 11. The one or more tangiblecomputer-readable storage media of claim 8, wherein the first imagingmodality is computed tomography or magnetic resonance imaging.
 12. Theone or more tangible computer-readable storage media of claim 8, whereinthe first pose comprises a non-standing position and orientation of thepatient leg relative to the patient foot.
 13. The one or more tangiblecomputer-readable storage media of claim 8, wherein the second posecomprises a standing position and orientation of the patient legrelative to the patient foot.
 14. The one or more tangiblecomputer-readable storage media of claim 8, wherein the first patientbone data and the second patient bone data are the results of twodifferent imaging events.
 15. A system for processing patient data, thesystem comprising: a network interface configured to receive one or moresets of patient data; a processing device in communication with thenetwork interface; and a computer-readable medium in communication withthe processing device configured to store information and instructionsthat, when executed by the processing device, performs the operationsof: receiving first patient data comprising at least one two-dimensional(2D) image of a patient leg and foot in a weighted pose; receivingsecond patient data comprising computed tomography (CT) images of thepatient leg and foot in a non-weighted pose, the first patient data andthe second patient data being the result of separate imaging events;generating a three-dimensional (3D) patient bone model of the patientleg and foot from the CT images, the 3D patient bone model comprising aplurality of 3D bone models representing individual bones of the patientleg and foot; and rearranging the plurality of 3D bone models to mimicthe weighted pose of the patient leg and foot in the at least one 2Dimage by: generating a plurality of 2D projections of poses of theplurality of 3D bone models, wherein the plurality of 2D projections ofposes that are generated are limited to only such poses that arepermissible given bio-kinematics of bones making up the plurality of 3Dbone models; comparing the plurality of 2D projections to contour linesoutlining perimeters of bones of the patient leg and foot in the atleast one 2D image; identifying particular 2D projections from theplurality of 2D projections that best-fit a shape and size of thecontour lines; and arranging the plurality of 3D bone models accordingto orientations represented by the particular 2D projections that wereidentified, and arranging the plurality of 3D bone models relative toeach other according to bone spacing in the weighted pose.
 16. Thesystem of claim 15, wherein the computer-readable medium incommunication with the processing device is configured to storeinformation and instructions that, when executed by the processingdevice, further performs the operations of: identifying landmarks in the3D patient bone model; building coordinate frames for the plurality of3D bone models of the 3D patient bone model from the identifiedlandmarks; identifying permissible rotational and translationalorientations between the coordinate frames of the plurality of 3D bonemodels; and providing a plurality of orientations for the plurality ofbone models within permissible rotational and translational orientationsfor which to limit the generation of poses given the bio-kinematics ofthe bones making up the plurality of 3D bone models.
 17. The system ofclaim 15, wherein the at least one two-dimensional 2D image comprises atleast one X-ray image.
 18. The system of claim 15, wherein thecomputer-readable medium in communication with the processing device isconfigured to store information and instructions that, when executed bythe processing device, further performs the operations of: planning atotal ankle replacement procedure.